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Executive Summary 


This final report presents findings from a multi-year evaluation of the Comprehensive 
Technical Assistance Centers, a federally funded program that provides technical assistance to 
states in connection with the Elementary and Secondary Education Act, as reauthorized by the 
No Child Left Behind (NCLB) Act of 2001. The law authorizing the Centers, the Educational 
Technical Assistance Act of 2002, mandated that a national evaluation of the program be 
conducted by the Institute of Education Sciences (IES). The legislation indicated that the 
evaluation should “include an analysis of the services provided... [and] the extent to which each 
of the comprehensive centers meets the objectives of its respective plan, and whether such 
services meet the educational needs of State educational agencies, local educational agencies, 
and schools in the region.” The program evaluation was conducted by Branch Associates, Inc., 
Decision Information Resources, Inc., and Policy Studies Associates, Inc. 

With the redesign of the Center program, the primary focus of technical assistance was 
directed to states. In order to build states’ capacity for carrying out NCLB responsibilities, which 
include assistance to struggling school districts and schools as well as other areas of NCLB 
program administration, the Center program was designed to supply ongoing technical assistance 
in using research knowledge and promising practices. There are two types of Centers: 

■ Sixteen Regional Comprehensive Centers (RCCs) are responsible for providing 
ongoing technical assistance to states assigned to their region, working with a 
range of one to eight states per Center 

■ Five Content Centers (CCs) are expected to supply knowledge to RCCs and work 
with RCCs to assist states in the CC’s specialty area: Assessment and 
Accountability, Instruction, Teacher Quality, Innovation and Improvement, or 
High Schools 

Given this program design, the evaluation provides a description of Center operations. It also 
reports on assistance delivery and contributions to state capacity as judged by managers in state 
education agencies (SEAs), on quality as judged by panels of subject-matter experts, and on 
relevance and usefulness as judged by practitioners who participated in Center activities or 
received Center products. The evaluation data, collected annually, pertain to the Center program 
years 2006-07, 2007-08, and 2008-09, covering three of the five program years starting with the 
second year of program funding. 1 2 

■ The operations of the RCCs and CCs were consistent with the Center program 
design. RCCs and CCs assessed client needs annually to determine their technical 
assistance plans, with informal communications as the mode most commonly 
reported for 2008-09. The most common activity found in sampled RCC projects" 


1 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 
32583-94. The awards were subsequently extended. 

2 For the purposes of this evaluation, the team identified “projects” as a common level of aggregation of Center 
activities that would constitute units large enough for review and rating, but focused enough for coherence. A 
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was “ongoing consultation and follow up” (82, 93, and 91 percent of the sampled 
RCC projects in years 2006-07, 2007-08, and 2008-09, respectively), consistent 
with the charge to provide frontline assistance on an ongoing basis to states. In 
CC projects the most common activity was “research collections and synthesis” 
(more than 70 percent of sampled projects in each year), consistent with the CCs’ 
prescribed focus on synthesizing, translating, and delivering knowledge to RCCs 
and states. Across the three years studied, both RCCs and CCs were more 
involved in each other’s projects. Among sampled RCC projects, the percentage 
that included direct assistance from CC staff was 18 percent in 2006-07, 22 
percent in 2007-08 and 30 percent in 2008-09. The percentage of CC projects that 
included RCC direct assistance was 1 1 percent in 2006-07, 12 percent in 2007-08, 
and 38 percent in 2008-09. In addition, by 2008-09 all 16 RCCs reported 
receiving knowledge resources from CCs and all 5 CCs reported providing 
knowledge resources to RCCs. 

■ Centers addressed the most frequently cited state priority of “ statewide systems 
of support,” and an increasing number of state managers reported each year 
that Center assistance served their purposes. “Systems of support” consists of an 
infrastructure for the delivery of onsite assistance, and strategies and materials 
designed to help struggling schools and districts improve student performance. 
The most widespread NCLB -related priority for state managers was “statewide 
systems of support or school support teams,” which was identified as a major or 
moderate priority for technical assistance by more than 90 percent of managers, 
weighted, in each year. Of this group of state managers, more than 90 percent 
reported each year that the Centers delivered assistance related to this 
responsibility. “Systems of support” was not only the most widely reported state 
priority but also the topic addressed in more Center projects in each year than any 
other topic, according to the inventories compiled by the Centers (19 percent of 
all projects in 2006-07, 25 percent in 2007-08, and 21 percent in 2008-09, 
compared with 10 percent or fewer projects addressing any other topic). With 
each state weighted equally in the analysis, the proportion of state agency 
managers reporting that assistance from the Centers had “served the state’s 
purposes completely” rose from about one-third (36 percent) in 2006-07 to more 
than half (56 percent) in 2008-09. 

■ Center assistance was reported by state managers as having expanded state 
capacity in “ statewide systems of support,” which has been a predominant focus 
of Center assistance. Among state managers who reported statewide systems of 
support or school support teams as a state priority for technical assistance in 
2008-09, 82 percent credited Center assistance with a “great” or “moderate” 
expansion of state capacity in this area. In other areas of state responsibility 
identified by state managers to be a priority for technical assistance, the 
percentage reporting a great or moderate expansion of state capacity in 2008-09 
ranged from 77 percent (for research-based curriculum, instruction, or 


“project” was defined as a group of closely related activities and/or deliverables designed to achieve a specific 
outcome for a specific audience. 
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professional development in academic subjects) to 39 percent (for NCLB’s 
provisions on supplemental educational services and choice). 

■ On average across each of the three years, expert panels rated sampled project 
materials between “ moderate ” and “high” for quality, and project participants 
rated the sampled projects “high” for relevance and usefulness. Pro gram- wide 
average ratings, on a 5 -point scale with 5 at the high end, were 3.34 in 2006-07, 
3.51 in 2007-08, and 3.57 in 2008-09 for technical quality; 3.94, 4.08, and 4.15, 
respectively, for relevance; and 3.69, 3.95, 3.96, respectively, for usefulness. 3 In 
addition, the average quality rating was consistently higher among CC projects 
than RCC projects by more than one-half of a standard deviation while RCC 
ratings went up each year. 4 The average ratings of relevance were higher for RCC 
than CC projects in 2006-07 and 2007-08 although CC ratings went up each year; 
there were no consistent differences in the usefulness ratings between RCCs and 
CCs. 


The Comprehensive Centers Program 

In its authorization under the Educational Technical Assistance Act of 2002, the Center 
program was given an overall charge of supporting state and local NCLB implementation. The 
U.S. Department of Education (ED), using discretion provided in the legislation, established two 
major program features that differed from the design of Comprehensive Center programs under 
prior legislation: 5 

■ First, the primary focus would be on assisting states to carry out NCLB 
responsibilities and helping build state capacity to deliver assistance to schools 
and districts; ED specified that Centers could only work directly with districts or 
schools under special circumstances. 

■ Second, awards would be made to 21 Centers to establish two-tiers of technical 
assistance with 16 RCCs and 5 CCs. They were instructed to work as follows: 

■ Each RCC was charged with providing ongoing assistance directly to 

states in its region (“frontline assistance”), serving the needs of either one 


3 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 

4 All project-level differences described in this report (e.g., more, higher) reflect a difference of one-half of one 
pooled standard deviation between groups of projects. Using a metric derived from Cohen ( 1988), the evaluation 
team estimated Cohen's d (an estimate of the effect size defined as the difference in means divided by the pooled 
standard deviation) and adopted the logic of Cohen for what would be considered a moderate difference. For this 
study, inferential tests of statistical significance were not conducted to examine project-level differences in these 
non-probability samples. All participant-level differences described in this report reflect statistical test of 
significance with a criterion value of p<.05. 

5 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 
32583-94. 
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large state or a group of two to eight states and other jurisdictions. 6 The 
RCCs were also expected to deliver technical assistance to their assigned 
states, addressing the needs and building capacity of the states to assist 
their districts and schools. 

Meanwhile, each CC would work on a nationwide basis to provide in- 
depth knowledge of the content and research within a particular 
substantive area: Assessment and Accountability, Instruction, Teacher 
Quality, Innovation and Improvement, or High Schools. CCs would 
facilitate access to, and use of, existing research and practices. 

The absolute priorities for the two types of Centers indicated that they 
should work together: Regional Centers should draw information and 
resources from Content Centers as well as other sources; and Content 
Centers should both supply knowledge to Regional Centers and “work 
closely with Regional Centers to provide technical assistance to States.” 


Research Questions and Methods 

The research priorities for the evaluation were primarily driven by the statute and focused 
on the following key research questions: 

1. How did the Regional Comprehensive Centers and Content Centers operate as part of the 

Comprehensive Technical Assistance Center program? 

■ How did Centers develop, refine, and carry out their plans for technical 
assistance? How did they define their clients’ educational needs and priorities? 

■ What were the objectives of the technical assistance the Centers offered? What 
kinds of products and services were provided by the Centers? 

■ How did the Regional Comprehensive Centers and Content Centers coordinate 
their work? 

2. What was the performance of the Comprehensive Centers in addressing state needs and 

priorities? How did their performance change over the period of time studied? 

■ How did the Centers’ state clients define their needs and priorities? 

■ To what extent, as reported by states, did Center assistance expand state capacity 
to address underlying needs and priorities and meet the goals of NCLB? 


6 The nonstate jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto Rico, 
the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of 
Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the Marshall Islands, and Republic of Palau. 
Throughout this report, the term “state” will be defined to include the 50 states as well as these other jurisdictions. 
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■ To what extent did states rely on other sources of technical assistance besides the 
Centers? What were other sources of technical assistance that states used? How 
did the usefulness of Center assistance compare with the usefulness of assistance 
from other sources? 

3. To what extent was the assistance provided by the Centers of high quality, high relevance, 
and high usefulness? 

■ Did the quality, relevance, or usefulness of Center assistance change over the 
period of time studied? 

■ What was the variation in the quality, relevance, and usefulness of Center 
assistance across types of projects and participants? 


The evaluation gathered information annually on the Center program for the years 2006- 
07, 2007-08, and 2008-09 from six data sources in order to address the research questions above. 
Data sources included: 

■ Management plans. The evaluation reviewed these plans as a data source for 
each Center’s intended focus at the beginning of the year, drawing from the plans 
a list of topics as foci of Center objectives. 

■ Project inventory forms and cover sheets. Each Center completed an inventory 
of its work that grouped related activities and deliverables into “projects,” with 
the project defined as a group of closely related activities and/or deliverables 
designed to achieve a specific outcome for a specific audience. Projects were in 
turn classified by the Centers into major, moderate, and minor projects on the 
basis of the relative level of effort they reflected. The Centers also classified the 
projects, according to the topics addressed, into 22 topical categories. 7 At each 
stage, the evaluation team provided written guidance and training for inventory 
development, reviewed the Centers’ drafts, and clarified definitions as needed. 
For projects sampled for the evaluation, the Centers prepared “cover sheets” 
providing brief descriptions and contexts for the activities and resources included 
in the project. The evaluation team used the cover sheets as a data source for 
coding project activities and resources. 


7 The 22 topics were: components of effective systems of support for states, districts, and schools; data use or data- 
driven decision making; formative assessment; reading; adolescent literacy; mathematics; dropout prevention; high 
school redesign or reform; transition to high school; special education curriculum, instruction and professional 
development; special education assessment; English language learners;” highly qualified teacher” provisions of 
NCLB; teacher preparation and induction; teacher professional development; supplemental educational services; 
Response to Intervention; migrant education; Indian or Native American education; data management and 
compliance; assessment design; and parent involvement. In addition, projects that addressed none of these 22 topics 
were categorized as “other.” 



■ Center staff interviews. Using structured response categories, Center staff were 
asked about how they planned their programs of work; how their plans evolved 
during the program year; and what they offered to clients with respect to the 
topics addressed, the delivery modes used, and their sources for content expertise. 

■ Survey of senior state managers. SEA managers were surveyed about their 
state’s technical-assistance needs and what was provided by the Centers 
(including their RCC and any CCs with whom their state had worked). 

■ Expert panel review. The same sample of major and moderate projects was 
reviewed for quality by a panel of experts. Content experts were recruited and 
trained to use standard criteria to rate the technical quality of the sampled Center 
projects on the basis of a review of all project materials. 

■ Survey of project participants. A representative sample of clients who had 
participated directly in the evaluation’s purposive sample of major and moderate 
Center projects furnished descriptive information, through surveys, on the 
activities and resources that the project had delivered to them. These clients 
included individuals working at the state level who had participated in RCC or 
CC projects, and RCC employees who were among the clients of CC projects. 
They rated the relevance and usefulness of the sampled projects. 


Center Operations 

In designing the Center program, ED established structures and expectations for the 
functioning of the Centers. Key features of the design, emphasized in ED’s Notice Inviting 
Applications, were a requirement for needs assessment in consultation with clients, a focus on 
technical assistance with state responsibilities in school and district improvement, and the 
specialized roles of RCCs and CCs. The Centers’ work from 2006-07 to 2008-09 conformed to 
the program’s requirements in each of these respects. The barriers to technical assistance that 
Centers most often reported were staff turnover in state agencies and limitations on the CCs’ 
scope of work. 

A key expectation of the Centers was to organize their plans around the priorities and 
needs of client organizations. At the start of each program year, the Centers were required to 
deliver a management plan to ED outlining the program of technical assistance they planned to 
provide. Across years, Centers used a range of methods to assess needs and plan technical 
assistance with their clients. Among RCCs, there was a shift away from conducting surveys for 
needs assessment while maintaining frequent interaction with states as a means of learning about 
needs: all 16 RCCs reported assessing needs for 2008-09 through ongoing communication with 
state staff (an increase from 15 RCCs in 2006-07); 15 had a designated liaison to the SEA on 
staff in 2008-09 (up from 13 in 2006-07); 8 of the 16 conducted surveys (down from 11). Across 
years, all 5 of the CCs reported forming their work plans with RCC input acquired through 
ongoing communication; in 2008-09 all 5 CCs reported surveying RCCs (up from 4 of the 5 in 
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2006-07). In addition, all 5 CCs reported learning about state needs for 2008-09 through ongoing 
interaction with states as well as through communication with RCCs. 

Centers were expected to show responsiveness to needs and requests for technical 
assistance but might not be in a position to respond to every client request. In each year, more 
than half of the Centers reported that they had turned down a client request for assistance, a 
situation that was handled differently by RCCs and by CCs. The number of Centers that reported 
having turned down one or more requests was 12 of the 21 in 2006-07, 13 of 21 in 2007-08, and 
14 of 21 in 2008-09. Among Centers that declined any client request for assistance in 2008-09, 
RCCs most often reported substituting a different type of assistance (7 of the 10 RCCs that 
turned down work used this strategy), but none of the CCs reported doing so. The reason most 
commonly reported by RCCs was that a request fell outside their legitimate scope of work (5 
RCCs vs. 1 CC). CCs reported more concerns with the requests fitting the Center’s priorities (2 
of the 4 CCs that turned down work vs. 2 of the 10 RCCs) or the Center’s capacity (2 of the 4 
CCs that turned down work vs. 2 of the 10 RCCs). 

The Centers placed a priority on assistance with the state role in supporting improvement 
in struggling schools and districts. In every year of the evaluation, on the inventories completed 
by Centers that grouped their technical assistance activities into projects and categorized projects 
into 23 topics, the most common topic for all Center projects was “components of effective 
systems of support — state, district, school,” a topic that included but was not limited to statewide 
systems of support and school support teams. Among all projects on the Center’s inventories, 19 
percent in 2006-07, 25 percent in 2007-08, and 21 percent in 2008-09 addressed the topic of 
systems of support, which in each year was more than twice as many as any other topic. 

Although the two types of Centers each retained a focus on activities distinctly associated 
with the original program design, their ways of working became more similar over the years. The 
guidance given by ED through the Center grant competition and afterwards laid out a particular 
structure for the Centers’ work: RCCs would specialize in interactions with state clients while 
CCs would specialize in activities that required a content focus. The most common activity 
found in sampled RCC projects was “ongoing consultation and follow up” (82 percent in 2006- 
07; 93 percent in 2007-08; 91 percent in 2008-09); in CC projects, it was “research collections 
and synthesis” (74 percent in 2006-07, 85 percent in 2007-08, and 77 percent in 2008-09), while 
fewer RCC projects included this activity (53 or 54 percent in each year) (exhibit ES.l). In 2008- 
09, in a departure from past CC practice, a majority of sampled CC projects (62 percent) 
included ongoing consultation and follow-up. 
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Exhibit ES.1. Sampled RCC and CC projects by types of activities and 
resources, by year 



Percent of sampled RCC 
projects 

Percent of sampled CC 
projects 

Activities and resources 

2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

2006-07 

(n=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

Ongoing consultation and follow- 
up 

82% 

93% 

91% 

22% 

15% 

62% 

Research collections and 
syntheses 

54 

53 

53 

74 

85 

77 

Engagement of participants in 
project planning 

45 

63 

59 

30 

31 

27 

Training events 

43 

55 

59 

37 

50 

50 

Task force meetings and work 

50 

58 

56 

7 

8 

8 

Conferences 

27 

35 

40 

63 

42 

38 

Support development of a formal 
plan to implement a program or 
policy 

19 

32 

31 

7 

15 

23 


EXHIBIT READS: For the 2006-07 program year, 82 percent of sampled Center projects included ongoing 
consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. In addition to serving 
as resource material for the expert reviewers, these cover sheets were coded by the evaluation team. 


The delivery of technical assistance depended on the Centers working effectively with 
their clients. Both RCCs and CCs described the barriers they perceived as having impeded their 
assistance to states. Turnover in staff within state offices or intermediary units was reported by 
both types of Centers as a barrier to achieving their objectives in assisting states (10 of 16 RCCs 
and 3 of 5 CCs). Turnover at the leadership level was a reported barrier for 8 RCCs and 3 CCs. 
Three of the 5 CCs reported “a state’s most important priorities for assistance fell outside the 
Center’s scope of work,” as a barrier; they indicated that some states wanted help with topics that 
went beyond their own assigned substantive focus. 

Under the two-tiered Center program design, RCCs and CCs were expected to work 
together to serve state clients. Among RCC projects, 48 percent had a CC contribution (of 
materials, in-person assistance, or advice) in 2006-07, 32 percent in 2007-08, and 47 percent in 
2008-09. Among CC projects, the percent incorporating some RCC contribution was 37 percent, 
38 percent, and 42 percent across the years. The extent to which RCCs and CCs drew on the 
other as substantive partners in delivering assistance increased in 2008-09: the percent of 
sampled RCC projects in which CCs delivered technical assistance went up from 18 percent in 
2006-07 to 30 percent in 2008-09, and the percent of sampled CC projects in which the RCCs 
delivered technical assistance rose from 1 1 percent in 2006-07 to 38 percent in 2008-09. 









With 16 RCCs and 5 CCs all charged with working with the other type of Center, 
coordination varied across the different pairs of an RCC and a CC. For example, while 15 of the 
16 RCCs reported teaming up with at least one CC to provide technical assistance to states, 14 of 
them reported teaming up with one of the CCs but 7 of them reported doing so with another of 
the CCs. In addition, CCs were expected to provide assistance to RCCs, and the barrier most 
often reported by both types of centers to have impeded CCs’ achievement of their technical 
assistance objectives with RCCs was that “RCCs’ most important priorities for assistance fell 
outside the CC’s scope of work” (reported as a barrier by 7 of 16 RCCs and 4 of 5 CCs). 


Extent to Which Centers Addressed State Priorities 

The perceptions of senior managers in state education agencies, who are involved in 
identifying state needs and priorities for technical assistance, provide a relevant perspective on 
the outputs of the program. Because the Centers had a mandated focus on the states, the extent to 
which state managers perceived that Center technical assistance served state purposes is one way 
of gauging the program’s attainment of its objectives. Capacity building for states is also a focus 
of this evaluation, because it was prominent as a goal for the Comprehensive Centers program. 
The first priority for all Centers, articulated by ED in the Notice Inviting Applications, included 
“helping states build the capacity to help school districts and schools implement NCLB 

o 

provisions and programs.” 

An increasing percentage of state managers (weighted) over three years reported that the 
Centers’ technical assistance “served the state’s purposes completely” (exhibit ES.2). Thirty-six 
percent of the state managers, weighted, chose this response for 2006-07, 47 percent for 2007-08, 
and 56 percent for 2008-09. 

Among the managers who said their state’s purposes were not completely served, a larger 
proportion in each year reported that they wanted more interaction with the Centers. The percent 
of weighted state managers saying, “Center staff are not able to spend as much time working 
with the state as we would like” was 17 percent of those who did not say the state’s purposes 
were “completely” served in 2006-07. The corresponding figures for subsequent years were 27 
percent in 2007-08 and 43 percent in 2008-09. 8 9 (These respondents were 9 percent, 10 percent, 
and 16 percent, respectively, of all state managers, weighted.) 


8 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, tune 3). 70(106), 
32585. 

9 The percentage of state managers reporting that their state’s purposes were not completely served varied by year. 
Thus, for the follow-up question (“reasons the technical assistance has been less helpful than it might be”), 
comparison of percentages from year to year may include variation in responses over time as well as changes in 
respondents addressing the question. 
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Exhibit ES.2. Extent to which technical assistance from the Centers served 
state purposes, as judged by state managers, by year 


Percent of state managers (weighted) 


It did not serve the state's 
purposes 


100 

80 

60 

40 

20 

0 



It was a good start, but some 
important priorities were 
not addressed 


It was a good start 


It served the state's 
purposes completely 


2006-07 2007-08 2008-09 

(n=56) (n=57) (n=54) 


EXHIBIT READS: In 2006-07, 36 percent of state managers, weighted, reported that Center 
technical assistance served the state’s purposes completely. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally 
represented in instances where more than one manager from a state responded. 


State managers in a high proportion of states reported that Centers delivered assistance on 
“statewide systems of support or school support teams,” which was the most widespread priority 
among areas of technical assistance for state managers. In 2007-08 and 2008-09, more than 
90 percent of state managers, weighted, identified this area of state responsibility as a major or 
moderate priority for technical assistance (95 percent in 2007-08, 94 percent in 2008-09). Of this 
group of state managers that reported this priority 10 , more than 90 percent reported that the 
Centers delivered assistance related to this responsibility (94 percent in 2007-08, 91 percent in 
2008-09). 

Looking at state reported capacity building across areas of major or moderate state 
priority for technical assistance, the highest percentage of state managers reported Center 
assistance to have expanded their agency’s capacity to a “great” or “moderate” extent in 
statewide systems of support or school support teams * 11 (72 percent in of those who rated the area 


10 The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 
priority for technical assistance varied by year. Thus, for the follow-up question about the priority areas in which 
states received technical assistance from Centers, a comparison of the percentages from year to year may include 
variation in responses over time as well as changes in respondents addressing the question. 

1 1 Percentages are based on the state manager respondents who rated each area as major or moderate technical 
assistance priority, which varied by year. Thus, for the question about state capacity building, comparison of 
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as a state priority in 2007-08 and 82 percent in 2008-09) (exhibit ES.3). The next-highest in both 
2007-08 and 2008-09 was “development or dissemination of research-based curriculum, 
instruction, or professional development programs in academic subject(s)” (64 percent in 2007- 
08 and 77 percent in 2008-09). In both years, the lowest was “administration of supplemental 
educational services and choice provisions” (44 percent and 39 percent, respectively, of those 
who rated the area as a priority), which was an area rated as a priority by the fewest state 
managers (49 percent and 48 percent, respectively). 


Exhibit ES.3. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by state managers who rated the area as 
a major or moderate technical assistance priority, by year 



Percent reporting 
capacity expanded by a 
great or moderate extent 

Area of state responsibility under NCLB 

2007-08 

2008-09 

Statewide systems of support or school support teams 
(n=56, n=50) 

72% 

82% 

Policies and practices for English language learners 
(n=43, n=40) 

59 

73 

State assessment and accountability systems 
(n=42, n=39) 

57 

59 

Development or dissemination of research-based 
curriculum, instruction, or professional development 
programs in academic subject(s) 

64 

77 

(n=41, n=39) 

Assistance with educators’ use of assessment data 
(n=37, n=36) 

62 

61 

Monitoring compliance with NCLB requirements 
(n=35, n=30) 

57 

57 

Administering supplemental educational services (SES) 
and choice provisions (n=25, n=26) 

Communication with parents or the public 
(n=25, n=26) 

44 

39 

48 

50 


NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 
technical assistance priority, which varied by year. State managers who chose the response, “Does not 
apply, or not able to judge,” were included in the denominator of the percent calculation. 

EXHIBIT READS: For the 2007-08 program year, among the 56 state managers (weighted) who reported 
that technical assistance in statewide systems of support or school support teams was a major or moderate 
priority for their state, 72 percent reported that technical assistance received from the Centers expanded the 
state’s capacity in this area to a great or moderate extent. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 
instances where more than one manager from a state responded. 


percentages from year to year may include variation in responses over time as well as changes in respondents 
addressing the question. 
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State managers reported that the Centers were one of multiple sources that they used for 
technical assistance, but over time they turned to the Centers for more purposes. The purposes 
for which states used the Centers more than other sources in each of the three study years were 
“to plan the initial steps in solving a problem” (reported as a purpose for Center technical 
assistance by at least 60 percent of state managers in each year) and “to develop the skills of 
SEA or intermediate education agency staff’ (at least 55 percent of state managers in each year). 
In 2007-08 and 2008-09, Centers were reported by state managers to be the top source for two 
additional purposes: to help states complete tasks where they lacked resources (58 percent and 64 
percent, respectively) or expertise (49 percent and 53 percent, respectively). 


Ratings of Center Assistance 

To assess the technical assistance provided by the Center program, quality, relevance, 
and usefulness of a sample of Center projects were rated. All sampled projects were identified by 
the Centers as “major” or “moderate” in their level of effort, relative to other projects in the same 
Center. The projects were rated for technical quality by panels of experts with strong knowledge 
of the content or substantive focus of the specific projects they reviewed. Projects’ relevance and 
usefulness were rated by a sample of participants — state staff, intermediate agency staff, local 
educators working on behalf of the state, and RCC staff — who were the intended beneficiaries of 
the project and who had received at least some of the technical assistance the project provided. 
Quality was judged on three dimensions; relevance was assessed with eight survey items and 
usefulness with 11 survey items (exhibit ESA). Each overall measure (relevance, usefulness, or 

quality) was calculated as the mean of ratings assigned to each item. The item-level ratings 

12 

themselves were based on 5-point rating scales. 

Based on the ratings, Center technical assistance was rated higher on each measure in 
each successive year, with program- wide average ratings in 2008-09 falling in a range between 
“moderate” and “high” for quality, and around “high” for relevance and usefulness (exhibit 
ESA). On a scale of 1 to 5 with a 3 representing “moderate” and a 4 representing “high,” the 
program-wide average ratings for the sampled projects were 3.34 in 2006-07, 3.51 in 2007-08, 
and 3.57 in 2008-09 for technical quality, scored by panels of content experts. Program-wide 
average ratings for relevance, scored by participants, were 3.94 in 2006-07, 4.08 in 2007-08, and 
4.15 in 2008-09. Average usefulness ratings for the program were 3.69 in 2006-07, 3.95 in 2007- 
OS, and 3.96 in 2008-09, also scored by participants. 13 


12 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 
levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 
set of items within the three measures and the different contexts for the ratings (experts who underwent training for 
the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 
different subsets of project activities), the ratings across the three measures are not directly comparable. 

13 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 
to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 
and each project sampled from a Center contributed equally to the Center mean. 
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Exhibit ES.4. 


Quality, relevance, and usefulness items 


From expert panel scoring 


From project participant surveys 


Technical quality 


Relevance 


Usefulness 


Reviewers were directed to 
assign a score to each 
dimension and to include the 
basis for their ratings on the 
rating form, including the specific 
artifacts on which their score was 
based. The three dimensions 
are: 

a. Demonstrated use of the 
appropriate documented 
knowledge base - to include 
an accurate portrayal of the 
current state of information 
with prominence to those 
with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 
knowledge base to the 
products and services 
provided - materials are 
consistent with the 
best/accurate information 
available and the 
presentation adequately 
conveys the confidence of 
the information 

c. Clear and effective delivery - 
information is well organized 
and written and accessible to 
the intended audience for 
easy use 


Based on your experience, to 
what degree was this set of 
activities and resources relevant 
to your work, in each of the 
following respects? 

a. Addressed a need or problem 
that my organization faces 

b. Addressed an important 
priority of my organization 

c. Addressed a challenge that 
my organization faces related 
to the implementation of NCLB 

d. Provided information, advice, 
and/or resources that could be 
directly applied to my 
organization’s work 

e. Addressed our particular state 
context 

f. Addressed my organization’s 
specific challenges (e.g., 
policy environment, leadership 
capacity, budget pressures, 
local politics) 

g. Provided information, advice, 
and/or resources that could be 
used to guide decisions about 
policies, programs, or 
practices 

h. Highlighted the implications of 
research findings (or 
information about best 
practice) for policies, 
programs, or practices 


Based on your experience, to 
what degree was this set of 
activities and resources useful to 
you, in each of the following 
respects? 

a. Provided resources that were 
easy to understand and easy 
to use 

b. Employed an appropriate 
format (e.g., a work group, a 
conference, individual 
consultation, written products) 

c. Provided adequate opportunity 
to learn from colleagues in 
other states 

d. Included adequate follow-up to 
support the use of new 
information and resources 

e. Were timely 

f. Helped my organization solve 
a problem 

g. Helped my organization 
maintain or change a policy or 
practice 

h. Helped my organization take 
the next step in a longer-term 
improvement effort 

i. Provided my organization with 
information or resources that 
we will use again 

j. Helped my organization 
develop a shared expertise or 
knowledge-base 

k. Helped individuals in my 
organization to develop skills 
that they will use again 
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Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, 
by center type and by year 



Technical quality 

Relevance 

Usefulness 


2006- 

2007- 

2008- 

2006- 

2007- 

2008- 

2006- 

2007- 

2008- 


07 

OS 

09 

07 

08 

09 

07 

OS 

09 

All Centers (N=21) 

3.34 

3.51 

3.57 

3.94 

4.08 

4.15 

3.69 

3.95 

3.96 

All RCCs (N=16) 

3.21 

3.41 

3.52 

3.99 

4.18 

4.15 

3.71 

3.99 

3.94 

All CCs (N=5) 

3.73 

3.86 

3.72 

3.78 

3.96 

4.17 

3.65 

3.84 

4.01 

Difference of RCC 
and CC means 

-0.52 t 

-0.45 t 

-0.20 t 

0.21* 

0.22 t 

-0.02 

0.06 

0.15* 

-0.07 

Pooled standard 
deviation (all Centers) 

0.41 

0.41 

0.37 

0.34 

0.22 

0.24 

0.34 

0.23 

0.27 

Ratio of difference in 
means to pooled 
standard deviation 

- 1.28 

- 1.09 

- 0.55 

0.62 

1.00 

- 0.08 

0.18 

0.64 

- 0.26 


NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 
ratings for the three quality dimensions. A notation of f indicates that the difference in the mean ratings between the 
CCs and RCCs within that year is at least one-half of one pooled standard deviation in the rating. 


EXHIBIT READS: In 2006-07 among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 
relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 
ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 
ratings. 


Given that the RCC and CC roles and activity emphases differed, the evaluation looked at 
variation across Center types. The mean ratings for types of Centers, based on their sampled 
projects, showed the CCs with higher mean ratings than RCCs for the quality of their sampled 
projects in all three years although RCCs’ average quality ratings were higher in each successive 
year (exhibit ES.5). The RCCs had higher mean ratings than CCs for the relevance of their 
sampled projects in 2006-07 and 2007-08 although the average ratings of relevance for CCs went 
up each year. There were no consistent differences in mean ratings of usefulness across types of 
Centers. 

The evaluation also looked at the relationships between the three measures: quality, 
relevance, and usefulness. It was reasoned that the content experts rating quality and the 
participants rating relevance and usefulness might be better able to judge different aspects of a 
Center project. On this rationale, content experts rated the projects for their technical quality, and 
participants rated the projects for relevance and usefulness. An examination of the associations 
among the three dimensions was conducted by calculating correlation coefficients. 14 Such a 
statistic indicates the strength and direction of a linear relationship between two factors. A 
correlation coefficient can vary from positive 1.00 (indicating a perfect positive relationship), 


14 For this analysis, the evaluation team used Spearman’s rank order correlation, as this non-parametric rating is the 
appropriate statistical function to describe correlations between two variables where the values of the variables are 
not normally distributed and are on a scale (such as ratings). 
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through zero (indicating the absence of a relationship), to negative 1.00 (indicating a perfect 
negative relationship). If the correlation is statistically significant (p <.05), we can have strong 
(95 percent) confidence that what we calculated is not due to chance. 

In every year, ratings of quality were unrelated to ratings of relevance and usefulness, 
although relevance and usefulness ratings were highly correlated with each other within each of 
the three data collection years. The correlation coefficient for relevance and usefulness was 
+0.84 for 2006-07, +.79 for 2007-08, and +.83 for 2008-09. This indicates that the extent to 
which participants rated the projects as relevant was associated with how they deemed the 
project to be useful to their agency. These coefficients were all statistically significant at p<.05. 
On the other hand, the results indicated correlations ranging from -0.12 to +0.04 between quality 
and relevance, and from -0.09 to +0.07 between quality and usefulness. Because these 
coefficients are not statistically significant we cannot be sure that they are different from zero 
(no relationship). In other words, the extent to which a project faithfully reflected the knowledge 
base on a topic and provided appropriate caveats about the quality of its evidence was unrelated 
to the extent to which participants deemed that project relevant or useful to their agency. 

Given the variation in ratings across Centers, additional analyses were conducted to 
explore whether there were consistent patterns between ratings and the particular features of the 
projects. Such information could provide suggestions for possible program improvement if there 
were consistent relationships. Quality ratings in 2008-09 were higher for RCC projects that 
included CC contributions of materials or in-person help than projects that the RCCs completed 
without CC contributions (3.72 vs. 3.39), although this was not the case in earlier years. In 
addition, quality ratings were higher in 2008-09 for projects that had been reviewed by CCs 
(3.83 vs. 3.46) and by outside experts (3.73 vs. 3.42) for quality assurance as opposed to projects 
that had not been reviewed in each of these ways (a project-level feature that was studied only in 
that year of the evaluation). In other analyses of project-level variation, projects that differed 
from each other in the activities they encompassed or the topics they addressed did not show 
differences in ratings of quality, relevance, or usefulness that were consistent across the three 
years. 


On the other hand, more consistent differences were found in ratings of relevance and 
usefulness awarded to projects by different types of participants. Higher ratings were awarded by 
those participants who had been involved in determining the project goals or design than by 
participants not involved in this way, and by those who had spent more time in project activities 
(i.e., 6 or more days) as compared to participants who had spent five days or less (these 
differences were statistically significant, with p <.01 for both relevance and usefulness). For 
2007-08 and 2008-09, also, each type of Center targeted its assistance more successfully to 
participants who worked in one type of agency, compared with participants who worked in other 
types of agencies: specifically, RCC projects were rated higher by participants from SEAs than 
participants from intermediate or local education agencies or schools; CC projects were rated 
higher by RCC staff than by SEA staff (statistically significant differences, with p<.05 for both 
relevance and usefulness). 
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